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SUMMARY 


On-the-job  training  requires  considerable  independence  on  the  part 
of  the  trainee.  Unlike  a student  in  a classroom,  the  trainee  must  arrange 
information  resources  in  such  a way  that  he  can  learn  how  to  perform  his 
specific  task  without  wasting  valuable  time  reading  irrelevant  information. 
He  must  further  direct  this  learning  himself. 

A computer-based  aid  to  self-directed  learning  has  been  developed 
to  meet  this  need.  This  aids  system  is  implemented  on  the  PLATO  system 
and  uses  the  touch-panel  capability  of  the  PLATO-IV  terminal.  Students 
are  presented  with  a task  which  requires  complex  learning,  and  they  are 
given  considerable  information  — much  more  than  is  needed,  in  fact  -- 
to  attain  the  task.  The  aids  system  is  designed  to  allow  students  to 
break  down  their  task  into  a set  of  more  easily  attained  objectives,  to 
decide  when  information  is  relevant  to  their  objectives,  and  in  general 
to  monitor  their  progress  toward  achieving  the  task. 

The  complete  training  aid  is  quite  complex,  so  that  students  are 
trained  in  its  use  over  a number  of  sessions.  New  features  of  the  system 
are  introduced  in  alternate  sessions,  and  students  then  practice  with  the 
system  using  a new  learning  task.  This  task  in  each  case  requires  the 
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student  to  troubleshoot  or  debug  a simulated  device.  This  device  produces 
output,  some  of  which  is  defective,  and  the  student  is  required  to  locate 
the  faulty  component  by  examining  the  defective  output  and  by  reading  an 
on-line  "technical  manual"  for  the  device. 

A pilot  experiment  has  been  completed  to  allow  a formative  evaluation 
of  the  self-directed  aids  system.  Although  the  results  of  this  experiment 
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found  no  statistically  significant  differences  between  the  treatment  groups, 
they  suggested  directions  for  future  research. 
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A FORMATIVE  EVALUATION  OF  A COMPUTER-BASED  INSTRUCTIONAL 
SYSTEM  FOR  TEACHING  JOB-ORIENTED  READING  STRATEGIES 

I.  INTRODUCTION 

Learning  from  textbooks  differs  significantly  from  learning  in  real- 
world  situations  such  as  on-the-job  training.  Information  in  a textbook 
is  arranged  in  such  a way  that  a student  is  led  in  an  idealized  fashion  to 
build  on  earlier  knowledge.  That  is,  the  textbook  writer  first  presents 
elementary  information,  then  more  complex  information  (based  on  the  ele- 
mentary information),  then  still  more  complex  information,  and  so  on.  In 
the  real  world,  however,  information  is  not  so  neatly  arranged.  Complex 
concepts  are  frequently  encountered  before  the  more  elementary  concepts 
upon  which  they  are  based.  Worse  yet,  there  is  generally  far  too  much  infor- 
mation available,  much  of  it  totally  irrelevant  to  what  the  student  wishes 
to  learn.  Since  no  one  has  pre-arranged  and  pre-digested  the  information 
for  him,  the  student  trying  to  learn  in  a real-world  situation  must  take  on 
those  responsibilities  himself.  He  must,  that  is,  be  self-directed  in  his 
learning. 

Not  infrequently,  inexperienced  technicians  find  themselves  assigned 
to  jobs  in  which  they  have  to  maintain  equipment  or  systems  they  have  not 
seen  before  or  may  have  encountered  only  briefly  in  school.  They  have,  in 
such  conditions,  a strong  need  to  learn  more  about  these  devices,  using 
available  technical  documents  as  a source  of  information.  The  technical 
manuals  they  consult  may,  in  some  instances,  presuppose  prior  knowledge 
that  is  incomplete  or  partially  forgotten.  Thus,  the  technician  on  the 
job  may  have  a requirement  to  learn  information  at  several  lower  levels 
of  complexity  as  well  as  at  the  technical  manual  level  and  to  organize  a 
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sequence  of  acquisition.  The  information  he  needs  may  not  be  contained  in 
a single  place  in  any  document,  and  the  structure  of  the  document--table 
of  contents,  index,  and  so  on--may  not  help  him  locate  the  proper  infor- 
mation. Linder  these  circumstances,  technicians  who  are  not  self-directed 
might  try  to  read  the  entire  technical  document  from  cover  to  cover,  obviously 
wasting  valuable  time.  On  the  other  hand,  some  technicians  might  ignore  the 
information  resources  available  and  simply  begin  sticking  test  probes  into 
the  defective  equipment,  equally  obviously  wasting  time.  In  either  case, 
the  technician  would  benefit  from  knowing  some  techniques  of  being  self- 
directed,  of  determining  which  information  is  relevant  to  his  specific  task 
and  learning  only  that  information. 

This  paper  describes  initial  steps  toward  the  development  of  a 
training  system  to  help  people  faced  with  this  kind  of  complex  learning 
task.  Our  research  plan  calls  for  several  cycles  of  development  and 
testing  of  the  training  system.  In  this  report  we  discuss  our  first 
pass  at  the  development  of  such  a system.  The  training  system  is  de- 
scribed and  the  results  of  a pilot  experiment  on  the  effectiveness  of 
the  system  are  reported.  This  formative  evaluation  will  be  used  to 
revise  the  self-directed  learning  system  described  below.  The  revised 
system  will  be  tested  again  on  college  students  and  then  revised  for 
use  in  technical  training  contexts.  New  data  bases  appropriate  to  such 
contexts  will  be  created,  and  the  system  will  be  tested  in  this  context. 

A sunmative  evaluation  will  be  performed. 
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Computer-based  Aid  to  Self -directed  Learning^ 

Our  training  program  is  designed  to  teach  students  how  to  use  a 
computer-based  aid  to  self-directed  learning  that  has  been  developed 
in  our  laboratory.  A learning  task  is  presented  to  a student,  and  he 
is  given  considerable  information— too  much  information,  in  fact— to 
complete  his  task.  The  aids  system  is  designed  to  allow  the  student 
to  break  down  his  task  into  a set  of  more  easily  attained  objectives, 
to  decide  when  a chapter  of  the  technical  manual  is  relevant  to  his 
objectives,  and  in  general  to  keep  track  of  his  learning.  This  aids 
system  can  be  thought  of  as  consisting  of  a number  of  "pages,"  each  of 
which  presents  certain  types  of  information  and  provides  the  user  with 
certain  options.  The  four  major  components  of  this  system  are  the  Task 
page,  the  Objectives  page,  the  Contents  page,  and  the  Relevant  Contents 
page.  (The  term  "page"  in  this  context  indicates  one  or  more  screen  dis- 
plays on  a PLATO-IV  panel).  From  any  of  these  pages,  the  student  can 
choose  to  go  to  any  one  of  the  others.  The  major  components  and  their 
subcomponents  are  shown  in  Figure  1. 

The  Task  page  states  the  overall  task  or  learning  goal  for  the  student. 
The  task  changes  for  each  session  that  the  student  uses  the  aids  system,  but 
in  each  case  it  involves  learning  enough  material  to  troubleshoot  a defective 
device  of  some  kind.  (See  Section  III  below).  The  Task  page  also  gives  the 
student  access  to  the  example  output  from  the  defective  device.  The 
student  uses  this  output  to  help  determine  the  source  of  the  fault  in  the 


We  thank  Steve  Cheney  for  advice  in  the  initial  stages  of  the  design 
of  the  aids  system  and  for  help  in  recruiting  students  to  test  early  ver- 
sions of  the  system. 
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device.  When  the  student  feels  he  is  ready  to  attempt  the  task,  he  can  go 
to  a test  accessible  from  the  Task  page.  This  test  requires  that  the  student 
identify  the  faulty  component  of  the  device;  if  he  fails  tliis  test,  he  is 
sent  back  to  the  aids  system  to  study  additional  material.  He  can  later 
return  to  the  Task  page  to  attempt  the  task  once  again.  Another  important 
function  accessible  from  the  Task  page  is  the  student's  goal  stack,  an  overt 
representation  of  dependencies  which  the  student  discovers  among  the  various 
parts  of  the  learning  process,  that  is,  task,  objectives,  and  information 
sources. 

The  goal  stack  the  student  sees  on  the  PLATO  terminal  screen  looks 
something  like  the  diagram  in  Figure  2.  The  arrows  in  the  goal  stack  dia- 
gram show  dependency  relationships  that  hold  among  the  student's  objectives 
and  the  information  resources  available.  For  example,  the  curved  line  from 
objective  1 to  objective  2 means  that  objective  2 cannot  be  attained  until 
objective  1 is  first  attained;  objective  2 is  thus  dependent  on  objective  1. 
Similarly,  the  line  from  information  source  7 to  objective  4 means  that 
objective  4 requires  the  understanding  of  information  source  7 for  its 
attainment.  The  curved  line  from  information  source  3 to  information  source 
1 means  that  1 is  dependent  on  3;  3 should  therefore  be  studied  before  1. 

The  student  is  taught  several  heuristics  to  help  him  use  the  goal 
structure  effectively.  For  example,  if  the  node  on  the  goal  tree  that  rep- 
resents a particular  goal  has  an  arrow  head  impinging  on  it,  then  that  goal 
should  not  be  attempted  until  the  goal  at  the  other  end  of  the  arrow  has  been 
attained.  This  is  a simple  restatement  of  the  principle  that  it  is  better 
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to  attempt  the  prerequisites  of  an  action  before  attempting  the  action. 

Since  the  goal  structure  keeps  a record  of  goals  attained  by  means  of 
check  marks  next  to  completed  goals,  this  rule  is  easy  to  heed. 

The  second  major  component  of  the  self-directed  learning  aid  is 
the  Objectives  page.  The  primary  function  of  this  page  is  to  maintain  a 
list  of  the  learning  objectives  based  on  the  task  at  hand.  From  the 
Objectives  page  the  student  can  formulate  new  objectives  that  he  or  she 
believes  are  necessary  to  the  accomplishment  of  the  task.  Once  the  student 
has  entered  an  objective,  it  will  be  listed  on  the  Objectives  page  whenever 
he  returns  to  that  page.  Two  other  functions  available  on  the  Objectives 
page  are  checking  off  objectives  that  have  been  attained  (by  reading  the 
relevant  information)  and  specifying  dependency  relationships  between  object- 
ives. When  the  student  utilizes  the  latter  option,  the  Aids  system  records' 
the  fact  that  there  is  a dependency  between  the  two  objectives  named  by  the 
student.  This  dependency  is  shown  whenever  the  student  chooses  to  look 
at  his  Goal  Stack  (accessed  from  the  Task  page),  and  an  arrow  is  drawn 
from  the  required  to  the  dependent  objective  on  the  Goal  Stack.  Thus,  the 
arrows  between  any  two  objectives  on  the  Goal  Stack  page  are  determined  by 
what  the  student  has  done  on  the  Objectives  page. 

The  Contents  page  simply  provides  a list  of  the  titles  of  "chapters" 
or  information  sources  of  the  technical  manual  that  covers  the  device  that 
the  student  is  troubleshooting.  The  student  can  scan  this  list  of  titles 
and  make  decisions  about  the  probable  relevance  to  his  objectives  of  some 
of  the  topics  mentioned.  When  he  decides  that  the  material  under  a certain 
title  is  likely  to  be  relevant  to  some  objective,  he  exercises  the  Choose- 
Title  option  from  the  Contents  page.  Picking  a title  has  the  effect  of 


throwing  control  immediately  to  the  Matching  page.  On  the  Matching  page, 
the  student  is  shown  the  list  of  objectives  and  the  title  he  just  picked; 
he  must  specify  which  of  those  objectives  requires  that  he  learn  the 
material  named  by  that  title.  If  he  does  not  want  to  match  any  of  his 
objectives  with  the  chosen  title,  then  he  must  cancel  his  choice  of  the 
title.  (If  he  wishes,  he  can  then  go  to  the  Objectives  page,  make  up  a 
new  objective,  and  then  return  to  the  Contents  page  to  select  the  title 
again,  planning  to  match  the  title  with  the  new  objective).  In  this  way, 
the  student  is  encouraged  to  select  only  those  titles  he  needs  to  solve 
his  problem.  As  a result  of  the  choices  made  on  the  Matching  page,  the 
Aids  system  remembers  which  of  the  chosen  information  sources  are  required 
by  which  of  the  objectives.  This  information  appears  whenever  the  student 
decides  to  look  at  his  Goal  Stack.  It  determines  the  arrows  that  are  drawn 
from  the  numbers  of  the  relevant  information  sources  to  their  objectives. 
(See  Figure  2). 


OBJECTIVES 


INFORMATION 

SOURCES 


TASK 


FIGURE  2.  A Sample  Goal  Stack 
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Choices  made  on  the  Contents  pages  have  one  other  consequence.  Those 
titles  that  are  chosen  as  relevant  to  some  objective  (and  are  matched  with 
the  objective)  will  appear  on  the  Relevant  Contents  page.  The  Relevant 
Contents  page  is  the  student's  personalized  table  of  contents  relevant  to 
the  troubleshooting  problem  he  is  trying  to  solve.  Whenever  the  student 
chooses  to  go  to  the  Relevant  Contents  page,  he  sees  a list  of  all  these 
chosen  titles.  A number  of  functions  are  available  from  this  page.  First, 
the  student  can  decide  to  read  any  of  the  information  sources  listed  there. 
Second,  if  a student  has  read  and  understood  an  information  source,  he  can 
check  off  the  title  on  the  Relevant  Contents  page  to  signify  that  this  sub- 
goal was  attained.  If  the  student  has  read  an  information  source  and  dis- 
covered that  it  was  irrelevant,  he  can  decide  to  remove  it  from  the  list  of 
relevant  information  sources.  A fourth  option  available  is  to  specify 
dependencies  between  information  sources.  For  example,  if  the  student  de- 
cides that  relevant  information  source  1_  cannot  be  understood  until  relevant 
information  source  2 has  been  understood,  then  he  can  specify  that  ]_  is 
dependent  upon  3^. 

The  last  three  choices  listed  above  all  have  consequences  for  the 
Goal  Stack.  If  the  student  has  checked  off  a title,  then  that  title's 
number  will  have  a check  mark  below  it  in  the  Goal  Stack.  If  a title  has 
been  removed  from  the  list  because  it  is  irrelevant,  then  it  will  not  appear 
in  the  Goal  Stack  at  all.  And  if  a dependency  between  two  information 
sources  has  been  specified,  then  an  arrow  will  connect  the  numbers  of 
their  titles  in  the  Goal  Stack. 

The  information  sources  or  "chapters"  themselves  are  quite  simple. 

Each  consists  of  a number  of  pages  through  which  the  student  can  progress. 
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The  student  can  page  through  the  information  source  either  forward  or 
backward.  In  addition,  from  any  page  the  student  can  elect  to  return 
to  the  Relevant  Contents  page. 
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II.  TRAINING  SEQUENCE 


Students  do  not  immediately  begin  with  the  complete  self-directed 
learning  aids  system  as  it  was  discussed  in  the  previous  section.  Instead, 
they  are  led  to  that  version  in  a series  of  training  sessions,  each  one 
having  more  of  the  features  discussed  in  Section  I than  the  previous 
session.  The  complete  training  sequence  is  shown  in  Figure  3. 


Session 


1. 


2. 


3. 

4. 

5. 


6. 


7. 


FAMILIARIZATION  WITH  TERMINAL 
INTRODUCTION  TO  SIMPLIFIED  AIDS  SYSTEM 


PRACTICE  WITH  SIMPLIFIED  AIDS  SYSTEM 


INTRODUCTION  TO  MORE  COMPLEX  AIDS  SYSTEM 
(WITH  GOAL  STACK) 


PRACTICE  WITH  MORE  COMPLEX  AIDS  SYSTEM 


INTRODUCTION  TO  FINAL  AIDS  SYSTEM 


PRACTICE  WITH  FINAL  AIDS  SYSTEM 


POST-TEST  WITH  FINAL  AIDS  SYSTEM 
(NEW  TASK  DOf^IN) 


FIGURE  3.  Training  Sequence  for  Use  of  Self- Directed  Aids  System 
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The  initial  session  familiarizes  the  student  with  the  PLATO  terminal 
and  introduces  him  to  the  aids  system.  The  student  begins  by  playing  a 
few  games  of  tic-tac-toe  against  the  computer  to  introduce  him  to  the 
idea  of  touching  the  terminal  panel.  He  then  goes  through  a training 
lesson  that  teaches  him  to  use  the  most  rudimentary  version  of  the  aids 
system.  (This  version  differs  from  the  full  system  discussed  in  Section  I 
in  the  following  ways:  first,  the  student  cannot  formulate  his  own  objec- 
tives, but  must  make  use  of  a set  of  objectives  provided;  second,  the  sys- 
tem does  not  provide  a Goal  Stack;  third,  the  Objectives  page  does  not 
provide  the  option  of  making  explicit  the  dependencies  among  the  objectives; 
fourth,  the  Relevant  Contents  page  does  not  provide  the  option  of  making 
explicit  dependencies  among  chosen  titles.)  This  first  session  has  three 
parts.  In  the  first  part,  the  student  is  taught  about  the  overall  structure 
of  the  Aids  system  and  is  given  a quiz  on  his  understanding  of  the  system. 
Those  students  who  score  below  the  criterion  must  repeat  this  section  of 
the  lesson.  In  the  second  part  of  the  lesson,  the  student  is  taught  about 
the  specific  functions  of  each  of  the  components  of  the  system.  This  section 
of  the  training  requires  that  the  student  step  through  each  of  these  func- 
tions in  a simulation  of  their  actual  use.  In  the  third  part  of  this 
session,  the  student  has  his  first  opportunity  to  practice  with  the  limited 
aids  system  on  a very  simple  task  (learning  to  use  the  PLATO  keyboard  to 
type  and  edit  answers). 

In  the  second  session  students  are  required  to  solve  a troubleshooting 
problem  through  the  use  of  the  simplified  Aids  system  they  learned  about 
in  the  first  session.  The  task  is  quite  complex,  and  most  students  require 
from  one  to  two  hours  to  accomplish  it. 
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The  third  session  introduces  students  to  a more  complex  Aids  system. 

To  the  simplified  system  they  have  already  learned  about,  the  Goal  Stack 
is  added.  In  addition,  the  options  to  specify  dependencies  among  objectives 
(on  the  Objectives  page)  and  to  specify  dependencies  among  information 
sources  (on  the  Relevant  Contents  page)  are  included.  The  lesson  requires 
the  student  to  make  appropriate  responses  in  a simulation  of  the  functions 
of  these  new  options. 

»■ 

In  the  fourth  session,  the  students  practice  with  this  more  complex 
Aids  system.  They  are  required  to  troubleshoot  the  same  type  of  device 
that  they  have  already  had  a troubleshooting  problem  on,  but  the  problem 
and  its  symptoms  are  new. 

The  fifth  session  introduces  the  student  to  the  writing  of  his  own 
objectives  on  the  automated  Aids  system.  When  this  lesson  has  been  com- 
pleted, the  student  has  been  introduced  to  the  complete  Aids  system  depicted 
in  Figure  1.  This  lesson  is  quite  short  and  is  usually  combined  with  that 
of  the  sixth  session  for  one  long  session. 

In  the  sixth  session,  the  student  practices  with  the  complete  Aids 
system.  The  new  troubleshooting  task  is,  again,  on  the  same  type  of  device 
as  were  all  the  previous  tasks. 

The  seventh  session  is  a post-test  session,  although  from  the  student's 
point  of  view  it  is  simply  another  practice  session  with  the  full  Aids  sys- 
tem. In  this  session,  the  troubleshooting  task  is  on  a defective  device  of 
a different  type  from  that  with  which  the  student  is  familiar.  New  informa- 
tion resources  are,  of  course,  provided. 
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III.  TASK  DOMAINS  FOR  THE  TRAINING  SYSTEM 


Each  time  a student  practices  with  some  version  of  the  Aids  system,  he 
must  solve  a complex  learning  task.  In  each  case,  the  task  is  to  trouble- 
shoot or  debug  a defective  device.  This  device  produces  output,  some  of 
which  is  incorrect;  by  examining  this  output  and  by  reading  information 
sources  on  the  various  components  of  the  device,  a student  can  determine 
which  component  is  faulty  (that  is,  causing  the  improper  output).  Each 
practice  session  with  the  Aids  system  has  a different  task— a different 
component  is  faulty,  and,  therefore,  different  symptoms  are  presented, 
each  time. 

Two  such  devices  were  selected  as  task  areas,  a sentence  generator 
and  an  essay  generator.  (These  devices  are  simulations,  not  physically 
embodied  machines.)  The  sentence  generator  was  used  in  sessions  3 through 
6,  and  the  essay  generator  was  used  in  session  7,  as  a post-test  of  the 
training.  These  task  domains  were  chosen  to  conform  to  a number  of  cri- 
teria: 

(1)  The  topic  matter  permits  the  construction  of  "debugging" 
or  troubleshooting  problems.  This  is  important  because  the  topic  matter 
is  to  be  analogous  to  the  electronics  troubleshooting  problems  that  con- 
front Naval  electronics  technicians. 

(2)  The  topic  matter  is  sufficiently  difficult  that  it  could 
not  be  easily  and  completely  comprehended  by  a single  reading  of  a simple 
"technical  manual"  (the  information  sources).  Again,  this  feature  is  an 
important  part  of  the  analogy  to  learning  about  the  maintenance  of  elec- 
tronic equipment. 
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(3)  The  topic  matter  is  sufficiently  simple  that  no  special 
technical,  scientific,  or  mathematical  skills  or  knowledge  are  prerequi- 
site to  an  understanding  of  the  "technical  manual."  This  feature  is  an 
important  concern  for  the  recruitment  of  subjects.  Ideally,  a large  class 

of  subjects  should  be  available  for  whom  the  task  topic  is  equally  unfamiliar. 

(4)  The  topic  matter  is  one  with  which  the  investigators  are 
sufficiently  familiar  that  they  can  easily  prepare  suitable  technical 
documents. 

The  Sentence  Generator 

The  major  components  of  the  sentence-generation  device  are  shown  in 
Figure  4.  A given  component  is  comprised  of  a series  of  sub-components. 

Arrows  in  the  diagram  show  the  flow  of  control  in  the  device.  Where  there 
are  choice-points  in  the  production  of  a sentence,  this  is  represented 
by  the  use  of  switches  in  the  diagram.  For  example,  within  the  Noun-Phrase 
Generator,  there  is  a three-way  choice  among  three  sub-components  of 
Noun-Phrase.  These  are  called  NPl , NP2,  and  NP3.  Only  one  of  these  serves 
as  the  activation  of  the  Noun-Phrase  Generator  at  one  time.  Within  the 
component  called  NP3  there  are  more  switches  signifying  other  options  in 
the  production  of  a noun  phrase  with  this  Noun-Phrase  Generator.  If  the 
NP3  unit  is  activated,  then  the  DET  unit  must  function;  the  Modifier-Phrase 
Generator  either  may  or  may  not  be  called  upon.  A dashed-line  box  surround- 
ing a component  (such  as  the  NP3  within  NPl,  or  the  NP  within  the 
Prepositional-Phrase  Generator)  signifies  that  control  is  surrendered  to  that 
component  (defined  elsewhere  in  the  diagram)  at  that  point.  When  that 
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NOUN- PHRASE  GENERATOR  VERB- PHRASE  GENERATOR 


ce  Ge 


embedded  component  has  finished  running,  control  returns  to  the  exit 
point  of  the  dashed-line  box.  The  presence  of  the  embedded  Noun-Phrase 
Generator  components  in  the  Noun-Phrase  Generator,  and  that  of  the  embedded 
Sentence  Generator  within  the  Verb-Phrase  Generator  makes  the  device  in- 
definitely recursive.  The  two  optional  components  after  the  Verb-Phrase 
Generator  permit  the  application  of  the  Dative-Shift  and  Passive  trans- 
formations to  the  output  of  the  rest  of  the  Sentence  Generator. 

The  diagram  in  Figure  4 is  a functional  analogue  to  a set  of  produc- 
tion rules  which  generate  hierarchical  structures,  plus  two  transformational 
rules.  The  rules  equivalent  to  the  component  diagram  are  given  below  (see 
next  page).  These  rules  describe  a powerful  device  which  produces  a wide 
variety  (although  not  all)  of  the  grammatical  sentence  types  in  English. 

It  provides  a rich  area  for  troubleshooting  or  debugging  problems. 

A student  assigned  to  troubleshoot  the  Sentence  Generator  has  access 
to  a technical  document  containing  25  chapters,  each  several  pages  long; 

10  of  these  chapters  discuss  various  aspects  of  sentence  grammar  but  are 
not  relevant  to  the  Sentence  Generator  itself.  The  remaining  15  chapters 
describe  the  functions  and  interrelationships  of  the  components  of  the 
Sentence  Generator.  A list  of  all  titles  of  these  information  resources 
is  given  below  (Page  18). 
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Phrase-Structure  Rules 


1. 

S — 

NP  + VP 

2. 

NP  — ; 

[nP^,  NP2,  NP3} 

3. 

NP^  — 

NP3+  Relpro  + 

VP 

4. 

NP2 

PN 

5. 

NPa-^ 

Det  + (MP)  + N 

+ (PP) 

6. 

MP  — 

(Adv)  + Adj 

7. 

PP  — 

P + NP 

8. 

VP  — ■ 

jvP^,  VP2,  VP3, 

9. 

VPi  ^ 

''i 

10. 

VP2- 

V^  + NP 

11. 

Vd  + NP  + Datprep  + NP 

12. 

VP4  - 

V^  + Comp  + S 

Transformational  Rules 

1. 

Passive 

SD:  NP  - V^  - NP 

1 2 3Z^ 

SC:  3,  was  + 2 + en,  by  + 1 

2. 

Dative 

Shift 

SD:  Vj  + NP  + ;^  + NP 

1 2 3 4 Zj> 

SD:  1,  4,  2,  0 

Symbols 

S = sentence,  NP  = noun  phrase,  VP  = verb  phrase,  Relpro  = relative  pronoun, 

PN  = Proper  noun,  Det  = determiner,  MP  = modifier  phrase,  N = noun,  PP  = 
prepositional  phrase,  Adv  = adverb,  Adj  = adjective,  Vi  = intransitive  verb, 

Vt  = transitive  verb,  Vd  = double  transitive  verb,  Datprep  = dative  preposition, 
Vc  = complementizing  verb.  Comp  = complementizer,  en  = perfective  marker. 
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INFORMATION  RESOURCES  AVAILABLE  FOR 
SENTENCE  GENERATOR  TROUBLESHOOTING 

1.  The  Dative-Shift  Component 

2.  Dependency  Grammar 

3.  The  EQUI-NP  Transformation 

4.  Finite-State  Grammar 

5.  Linear  and  Hierarchical  Structure 

6.  The  Mod-Phrase  Generator 

7.  The  Noun-Phrase  Generator 

8.  The  NPl  Component 

9.  The  NP2  Component 

10.  The  NP3  Component 

n.  The  Particle  Movement  Transformation 

12.  The  Passive  Component 

13.  Phrase-Structure  Grammar 

14.  The  Prep-Phrase  Generator 

15.  Rearrangement  Transformations 

16.  The  Sentence  Generator 

17.  Syntactic  Trees 

18.  The  Transformational  Component 

19.  Transformational  Grammar 

20.  The  Verb  Deletion  Transformation 

21 . The  Verb-Phrase  Generator 

22.  The  VPl  Component 

23.  The  VP2  Component 

24.  The  VP3  Component 

25.  The  VP4  Compoiient 
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The  technical  document  contains  a list  of  grammatical  sentences 
generated  by  a fully-functioning  device.  (In  addition,  the  discussion 
of  each  component  contains  representative  examples  of  phrases  or  words 
correctly  generated  by  that  component.)  As  examples  of  sentences  gener- 
ated by  the  device  in  Figure  4,  consider  the  list  below: 

The  quarterback  passed  the  ball  to  the  tight  end. 

Professor  Hotchkiss  is  sleeping. 

A very  naive  freshman  bought  the  wrong  book. 

The  teaching  assistant  who  gave  the  coed  an  A+ 
was  visited  by  the  dean. 

Harry  thinks  that  Frank  graduated. 

A student  in  the  back  row  coughed. 

The  instructor  realized  that  the  students  were  snoring. 

Other  chapters  of  the  technical  document,  which  discuss  other  com- 
ponents of  the  sentence  generator,  also  present  lists  of  sample  possible 
outputs  for  those  components.  For  example,  the  "Noun-Phrase  Generator" 
chapter  lists  a sampling  of  grammatical  noun  phrases,  such  as 
the  tight  end 
a very  naive  freshman 
Harry 

the  teaching  assistant  who  gave  the  coed  an  A+ 

When  subjects  are  presented  with  their  task,  they  see  a similar  list 
of  sentences,  but  some  of  these  sentences  are  ungrammatical  due  to  the 
failure  or  malfunction  of  a particular  component  of  the  sentence-generating 
device.  For  example,  consider  the  following  list.  (An  asterisk  (*)  before 
a sentence  indicates  that  it  is  ungrammatical.) 
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*The  textbook  was  written  by. 

The  dean  sent  a letter  to  the  department  heads. 

A student  who  failed  the  exam  is  crying. 

That  extremely  young  freshman  surprised  the  professor. 

*'The  secretary  in  the  chairman's  office  discovered  that 
taught  yoga. 

The  trophy  was  presented  to  the  team  by  the  chancellor. 

(English  speakers  are  sometimes  clever  enough  to  provide  a semantic  interpre- 
tation of  a sentence  marked  as  ungrammatical.  The  point  is  that  the  sentence 
is  nonetheless  ungrammatical  according  to  the  grammar/device  given  to  the 
student.)  In  this  case,  the  defective  component  is  NP2  (alternatively, 
phrase-structure  rule  4),  which  failed  to  output  proper  nouns.  One  way  of 
showing  the  nature  of  the  defect  is  presented  in  the  diagram  in  Figure  5. 

As  can  be  seen  in  this  drawing,  the  NP2  component  is  "empty."  It  has  no 
effect,  other  than  to  permit  exit  from  the  Noun-Phrase  Generator  without 
producing  a noun-phrase,  whenever  the  second  position  of  the  highest-level 
switch  in  the  Noun-Phrase  Generator  is  chosen.  When  the  student  correctly 
selects  NP2  as  the  faulty  component,  he  has  solved  his  task. 

The  Essay  Generator 

The  second  device,  an  Essay  Generator,  is  depicted  in  Figure  6.  The 
Essay  Generator  is  supposed  to  produce  well -formed  essays  on  a variety  of 
topics.  It  accomplishes  this  end,  in  theory,  by  the  sequential  activation 
of  a number  of  its  components.  A given  component  ordinarily  contains  a 
number  of  subcomponents.  Arrows  in  the  diagram  show  the  flow  of  control 
in  the  device.  Where  there  are  choice  points  in  the  production  of  an 
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FIGURE  5 


Diagram  of  a Faulty  Noun-Phrase  Generator 
Component  of  the  Sentence  Generator  Device. 
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PARAGRAPH  GENERATOR 


Ed 

g 

D 

O 

O 


oc 

Ed 

N 

M 

Z 

< 

O 

CJ 

M 

Ok 

o 

H 


hJ 

g 

6 OS 

M 

H o: 

M Ed 

8a 

<-)  N 

< Ed 

O N 

M M 

hd  2 

< 

H 

2 ^ 

CO 

z 

M < 

S 9 

< 

•d  o 

O Ctf 

o 

a.  p>i 
^ ° 

o 

1.  ■ - ■ J 

CO 


zr 


Ed 


-22- 


FIGURE  6.  Essay  Generator. 


essay,  this  is  represented  by  the  use  of  switches  in  the  diagram.  For 
example,  within  the  Topic  Organizer,  there  is  a three-way  choice  among  three 
subcomponents  of  the  Topic  Organizer.  These  are  called  the  Chronological 
Organizer,  the  Causal  Organizer,  and  the  Exemplification  Organizer.  There 
is  a four-way  switch  within  the  Paragraph  Generator,  the  setting  of  which 
determines  whether  a given  paragraph  will  be  produced  by  the  Claim-and- 
Evidence  Unit,  the  Event-and-Reactions  Unit,  the  Compare-Contrast  Unit,  or 
the  Princi pi e-and- Inference  Unit.  Students  assigned  to  accomplish  trouble- 
shooting tasks  based  on  the  Essay  Generator  have  access  to  a technical 
manual  of  109  pages  on  the  PLATO  system.  This  technical  manual  consists 
of  twenty-one  chapters,  fourteen  of  which  discuss  the  functions  and  inter- 
relationships of  the  components  of  the  Essay  Generator  and  contain  examples 
of  the  outputs  of  the  various  components  and  of  the  entire  system  when  it 
is  functioning  properly.  Which  of  these  chapters  are  relevant  depends, 
of  course,  upon  the  specific  troubleshooting  task  encountered  by  the  student. 
(The  other  seven  chapters  contain  general  information  about  writing  but  have 
nothing  to  do  with  troubleshooting  the  device;  thus,  these  chapters  are 
always  irrelevant.)  The  list  of  all  titles  of  the  information  sources  is 
given  on  the  next  page. 

An  example  of  the  output  of  the  Essay  Generator  when  it  is  functioning 
properly  is  given  below. 

Some  of  the  Effects  of  Watergate 

The  Watergate  scandal  generally  refers  less  to  the  actual 
break-in  at  Democratic  headquarters  than  to  the  later  attempts 
to  cover  up  White  House  involvement  in  the  planning  of  the 
operation.  The  discovery  of  this  involvement  and  subsequent 
widespread  publicity  had  a number  of  far-reaching  effects. 
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INFORMATION  RESOURCES  AVAILABLE  FOR 
ESSAY  GENERATOR  TROUBLESHOOTING 

1.  Causal  Organizer 

2.  Chronological  Organizer 

3.  Claim-and-Evidence  Unit 

4.  Compare-Contrast  Unit 

5.  Counter 

6.  Elaboration  Unit 

7.  Essay  Generator 

8.  Event-and-Reactions  Unit 

9.  Exemplification  Organizer 

10.  Figures,  Graphs,  and  Illustrations 

11.  Footnotes 

12.  Headings 

13.  Paragraph  Generator 

14.  Parts  of  Speech 

15.  Principle-and-Inference  Unit 

16.  Reference  Citations  in  Essays 

17.  Sentence  Characteristics 

18.  Sentence  Producer 

19.  Sentence  Types 

20.  Topic  Organizer 

21.  Topic  Sentence  Unit 
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One  result  was  the  eventual  resignation  of  the  President 
of  the  United  States  while  under  threat  of  impeachment.  This, 
in  turn,  meant  a new  administration  with  a largely  new  cabinet. 

Another  effect  of  Vlatergate  is  that  the  public  has 
become  very  suspicious  of  its  elected  officials.  In  the 
last  election,  being  an  incumbent  or  having  political  experi- 
ence often  seemed  to  be  more  of  a liability  than  an  advantage 
to  a candidate.  Citizens  are  suddenly  quick  to  demand 
explanations  for  any  improprieties. 

A third  effect  has  been  a change  in  the  relative  strengths 
of  the  Republican  and  Democratic  parties.  The  Republicans  have 
lost  membership,  while  the  Democrats  have  gained.  The  Republican 
party  treasury,  which  had  had  a surplus,  is  now  in  the  red.  The 
Democratic  treasury,  by  contrast,  had  been  deeply  in  the  red  but 
has  since  almost  fully  recovered. 

When  subjects  are  presented  with  the  task  of  debugging  the  Essay  Generator, 
they  see  several  such  essays,  but  some  of  them  are  defective  due  to  the  failure 
or  malfunction  of  a particular  component  in  the  Essay  Generator.  The  nature 
of  the  defect  depends  upon  the  type  of  component  that  is  defective.  For 
example,  if  some  component  within  the  Topic  Organizer  is  faulty,  then  the 
paragraphs  within  an  essay  might  appear  in  a random  order  rather  than  the 
orders  specified  by  those  components.  If  the  defect  lies  in  a component  of 
the  Paragraph  Generator,  then  the  sentences  within  a paragraph  might  appear 
in  the  wrong  order.  For  example,  consider  the  following  essay. 


Questionable  "Scientific"  Theories 

Recently  a number  of  questionable  theories  have  been 
proposed  by  scientists  working  outside  their  areas  of  specialty. 
In  many  cases  these  theories  have  been  avidly  adopted  by  large 
segments  of  the  public.  Yet  scientists  in  the  fields  that 
deal  with  these  theories  are  often  skeptical  of  the  claims  made. 

The  public,  however,  has  responded  favorably  to  Professor 
Bandersnatch's  numerous  appearances  on  television  talk  shows 
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and  has  purchased  over  1.5  million  copies  of  his  book,  Pyrami d 
People.  Archaeologists  and  Egyptologists  in  particular  have 
greeted  his  claim  with  hoots  of  derision.  One  example  of  this 
phenomenon  was  the  reaction  to  Professor  Arnold  Bandersnatch's 
announcement  that  the  ancient  Egyptian  pyramids  are  actually 
the  remnants  of  ancient  spaceships  to  Earth. 

However,  since  the  appearance  of  Talmay's  book.  Hair  Oil , 
in  September  of  last  year,  sales  of  Vitamin  E in  this  country 
have  increased  150%.  Doctors  and  biologists  have  almost  uni- 
versally scoffed  at  this  idea.  Another  example  is  the  claim 
made  by  the  physicist  Elmer  Talmay  that  Vitamin  E,  taken  in 
large  doses,  will  prevent  hair  loss. 


In  this  case,  the  defective  component  is  the  Event-and-Reactions  Unit. 

All  paragraphs  of  this  type  have  scrambled  sentence  order,  with  reactions 
to  some  event  appearing  before  the  statement  of  that  event.  All  other 
paragraph  types  are  correct,  however. 
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IV.  COGNITIVE  MODEL  FOR  SELF-DIRECTED  LEARNING 


One  way  of  viewing  the  goals  of  this  research  is  to  say  that  we  intend 
to  find  the  means  to  teach  people  how  to  do  effective  web- learning  (described 
in  Norman,  1973,  1974,  in  press).  What  is  it  that  they  will  know  when  they 
have  graduated  from  our  training  procedures?  How  will  what  they  know  guide 
their  learning  of  complex  materials  in  the  future? 

Our  answers  to  these  questions  are  couched  in  terms  of  schema-theory 
(Norman,  Rumelhart,  & LNR,  1975;  Rumelhart  & Ortony,  in  press;  Munro  & 

Rigney,  1977).  The  central  tenet  of  schema-theory  is  that  knowledge  guides 
thought.  Stated  baldly,  this  seems  to  be  a truism.  In  schema-theory,  how- 
ever, explicit  claims  are  made  about  the  means  by  which  knowledge  guides 
thought.  Computer  simulations  of  schema-theory  models  provide  rigorous 
tests  of  the  adequacy  of  the  proposed  mechanisms  for  the  relation  of  con- 
cepts in  memory  (of  knowledge).  Knowledge,  in  turn,  to  a large  extent, 
consists  of  "frozen"  or  fossilized  activations— copies  of  other  concepts  in 
memory,  with  specific  details  determined  by  the  particular  contexts  within 
which  those  concepts  were  activated  (see  Munro  & Rigney,  1977,  for  further 
explanation) . 

In  schema-theory  terms,  the  knowledge  that  subjects  acquire  as  a result 
of  the  training  described  elsewhere  in  this  report  is  best  represented  in 
terms  of  a prescriptive  schema.  A prescriptive  schema  is  a conceptual  struc- 
ture, which,  when  activated,  gives  people  the  impression  that  they  are  giving 
themselves  instructions.  Prescriptive  schemata  are  responsible  for  the  effects 
that  we  attribute  to  "self-direction  strategies."  The  set  of  schemata  that 
students  acquire  from  our  training  program  is  an  abstract  conceptual  struc- 
ture with  considerable  scope.  (The  uses  of  the  terms  "abstractness"  and 
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"scope"  with  respect  to  schemata  are  discussed  in  Munro  & Rigney,  1977).  Here 
are  the  hypothesized  schemata  that  we  believe  students  acquire  as  a result 
of  their  training. 

(1)  SELF-DIRECTED-LEARNING  (TASK) 

is  when 

BUILD-GOAL-STRUCTURE  (TASK) 

TASK-PURSUE  (TASK) 
end. 

(2)  BUILD-GOAL-STRUCTURE  (TASK) 

i s when 

ANALYZE  (TASK,  for  OBJECTIVES  (TASK))^ 

PREREQUISITE-SEARCH  (for  EACH  (OBJECTIVE),  in  OBJECTIVES) 
PREREQUISITE-SEARCH  (for  EACH  (OBJECTIVE),  in  CONTENTS) 
end. 

(3)  TASK-PURSUE  (TASK) 

is  when 

EXAMINE  (GOAL-STRUCTURE) 

UNTIL  (CHECKED  (EVERY  (OBJECTIVE)),  PURSUE  (OBJECTIVE)) 

TASK-ATTEMPT  (TASK) 
end. 

(4)  TASK-ATTEMPT  (TASK) 

is  when 

IF  (DO  (TASK),  then  QUIT,  else  SELF-DIRECTED-LEARNING  (TASK)) 
end. 


2 

The  ANALYZE  sub-schema  has  not  yet  been  represented.  How  people  are 
able  to  discover  the  prerequisites  or  component  actions  of  a task  is  not 
well  understood. 
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(5)  PREREQUISITE-SEARCH  (for  GOALS,  in  SUBGOAL-SET) 

is  when 

FOR-EACH  (MEMBER,  of  SUBGOAL-SET, 

IF  (PREREQUISITE  (MEMBER,  for  GOAL), 

then  (SPECIFY-DEPENDENCY  (MEMBER,  to  OBJECTIVES-LIST) ) ) ) 

end. 

(6)  PURSUE  (GOAL)^ 

is  when 

FOR-EACH  (SUBGOAL  (NECESSARY  (SUBGOAL,  to  GOAL)),  in  GOAL-STRUCTURE, 
WHILE  (ANY  (UNSATISFIED  (SUBGOAL'  (NECESSARY  (SUBGOAL',  to 
SUBGOAL)))), 

PURSUE  (SUBGOAL')) 

TRIAL  (SUBGOAL)) 
end. 

(7)  UNSATISFIED  (GOAL) 

is  when 

NOT  (CHECKED  (GOAL)) 

NOT  (ELIMINATED  (GOAL)) 
end. 

(8)  TRIAL  (GOAL) 

is  when 

ATTEMPT  (GOAL)  to  ATTEMPT  (ACTION,  of  GOAL) 

EVALUATE  (GOAL) 
end. 


3 

This  structure  is  a variant  of  Rumelhart  & Ortony's 
for  TRYing,  a subschema  of  their  PROBLEM-SOLVING  schema. 


(in  press)  schema 
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(9)  EVALUATE  (GOAL) 
is  when 

IF  (NECESSARY  (GOAL,  to  HIGHER-GOAL), 

then  IF  (SATISFIED  (GOAL),  then  CHECK  (GOAL), 
else  TASK-PURSUE  (TASK)), 
else  ELIMINATE  (GOAL,  from  GOAL-STRUCTURE)) 
end, 

(10)  ATTEMPT  (GOAL) 
is  when 

IF  (BELIEVE  (CAUSE  (ACTION,  SATISFIED  (GOAL))), 
then  DO  (ACTION), 

else  when  SUCCEED  (PREREQUISITE-SEARCH  (for  GOAL)), 

ATTEMPT  (PREREQUISITE  (GOAL))) 

end. 

According  to  the  first  of  these  schemata,  the  student  believes  that  the 
way  to  achieve  a task  through  self-directed  learning  is  first  to  build  a goal 
structure  and  second  to  pursue  the  task,  using  that  goal  structure.  The 
second  schema  listed  above  describes  what  is  involved  in  building  a goal 
structure.  One  analyzes  a task  for  objectives  (subgoals  necessary  for  the 
performance  of  the  task),  then  one  searches  for  prerequisite  relationships 
among  these  objectives,  between  the  available  information  resources  and 
the  objectives,  and  among  the  relevant  available  information  resources. 
However,  the  schema  does  not  contain  explicit  reference  to  the  process  of 
adding  these  relationships  to  the  goal  structure,  because  the  goal  structure 
is  constructed  for  the  student  by  the  program  that  aids  him  or  her  in  self- 
directed  learning.  The  fifth  schema  listed  above  is  an  essential  part  of 
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the  goal-structure-building  schema,  since  it  specifies  how  the  search  for 
prerequisites  is  conducted. 

The  second  major  part  of  self-directed  learning,  after  building  a goal 
structure,  according  to  the  above  schemata,  is  to  pursue  the  task.  The 
third  schema  above  gives  the  top-level  structure  for  task  pursuit.  One 
examines  the  newly  constructed  goal  structure  first;  then  one  pursues  the 
objectives  included  in  that  goal  structure  until  every  one  of  them  has  been 
checked.  (Checking  is  the  process  by  which  a student  marks  the  attainment 
of  a subgoal,  using  the  aids  program  on  PLATO).  When  all  the  necessary 
objectives  have  been  checked,  the  student  attempts  the  task.  If  the  attempt 

fails  (see  schema  #4),  then  he  begins  the  self-directed  learning  process 

again,  reconstructing  or  modifying  the  goal  structure. 

The  pursuit  of  objectives  is  governed  by  the  sixth  schema  given  above. 
This  is  a recursive  procedure  that  traces  down  dependency  relationships  in 
the  goal  structure.  When  a goal  is  found  that  has  no  prerequisites,  that 

goal  is  subjected  to  a trial.  This  means  (see  #8,  9,  & 10)  that  the  student 

does  an  action  to  bring  about  the  goal  and  then  evaluates  the  results  of 
that  action.  If  the  goal  is  satisfied,  he  checks  the  goal  and  then  pops 
back  to  the  appropriate  point  in  the  procedure  that  is  pursuing  an  objec- 
tive. If  it  is  not  satisfied,  he  looks  for  a new  way  to  pursue  his  over- 
all task.  If  the  attempt  reveals  that  the  goal  was  unnecessary  to  the 
attainment  of  its  higher  goal,  then  it  is  dropped  from  the  goal  structure. 

The  above  schemata  constitute  working  hypotheses  about  the  nature  of 
the  conceptual  changes  brought  about  by  training  in  the  self-directed 
learning  aids  program  discussed  above. 
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The  prose  explanations  of  these  schemata,  above,  emphasize  the  way  in 
which  these  schemata  call  each  other  in  a top-down,  conceptually-driven 
processing  mode.  Naturally,  there  is  also  a bottom-up,  data-driven  aspect 
to  the  activation  of  these  schemata  in  normal  circumstances.  For  example, 
when  a student  finds  that  he  has  satisfied  a goal  (say,  as  a result  of 
reading  one  of  the  relevant  information  resources),  this  activates  the  sub- 
schemata in  the  fourth  line  of  the  ninth  schema  presented  above.  The 
activation  of  these  subschemata  (IF  (SATISFIED  (GOAL),  then  CHECK  (GOAL,...) 
activates,  in  a data-driven  fashion,  its  "parent"  schema,  EVALUATE.  The 
activation  of  EVALUATE,  in  turn,  can  activate  the  schema  that  calls  i_t,  and 
so  on,  so  that  activation  spreads  in  an  upward  as  well  as  a downward  direc- 
tion. 
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V.  EXPERIMENT 


An  experiment  was  conducted  to  test  the  effects  of  the  self-directed 
learning  aids  system.  A control  condition  was  established,  containing  only 
the  Task  and  Contents  pages  of  the  system  described  in  Section  II,  A student 
in  the  control  condition  has  the  same  learning  task  and  the  same  information 
to  read,  but  he  has  none  of  the  Aids  system  available  to  a student  in  the 
experimental  condition.  (Information  sources  in  the  control  condition 
are  accessed  directly  from  the  Table  of  Contents,  As  soon  as  the  student 
touches  a title,  he  is  shown  the  corresponding  information  source). 

Control  Training  Sequence 

The  training  sequence  for  control  subjects  is  similar  to  that  for  experi- 
mental subjects,  except  that  the  basic  system  is  never  modified  for  them,  so 
that  there  is  no  need  for  teaching  sessions  other  than  the  initial  one. 
Consequently,  all  sessions  are  practice  sessions  using  the  control  system. 

The  complete  sequence  iS  shown  in  Figure  7. 


■ 

FAMILIARIZATION  WITH  TERMINAL 
INTRODUCTION  TO  CONTROL  AIDS  SYSTEM 

2 

PRACTICE  WITH  CONTROL  AIDS  SYSTEM 

3 

PRACTICE  WITH  CONTROL  AIDS  SYSTEM 

m 

PRACTICE  WITH  CONTROL  AIDS  SYSTEM 

5 

POST-TEST  WITH  CONTROL  AIDS  SYSTEM 
(NEW  TASK  DOMAIN) 

FIGURE  7".  Training  Sequence  for  Control  Subjects 
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The  initial  session  begins  in  the  same  way  as  in  the  experimental  condi- 
tion, with  a session  in  which  the  student  is  first  given  some  practice  using 
the  touch  panel  of  the  PLATO  terminal  by  playing  tic-tac-toe.  This  is  followed 
by  a two  part  PLATO  lesson  on  the  functions  of  the  control  "Aids"  system. 

As  with  the  experimental  group  students,  each  part  of  this  lesson  is  followed 
by  a quiz  which  the  student  must  pass  in  order  to  progress.  This  introduction 
is  followed  by  a short  practice  session  using  a very  simple  learning  task. 

In  the  second,  third,  and  fourth  sessions,  the  student  solves  complex  trouble- 
shooting problems  (one  for  each  session)  using  the  control  "Aids"  system. 

Each  of  these  tasks  involves  a different  problem  with  the  same  type  of  device, 
the  Sentence-Generator.  These  sessions  provide  practice  for  the  student  in 
the  use  of  the  control  "Aids"  system  and  in  troubleshooting  problems  on 
devices  of  the  sort  used  for  these  exercises.  In  the  post- test  (Session  5), 
students  are  to  use  whatever  learning  skills  they  acquired  during  their 
training  to  perform  a troubleshooting  task  in  the  new  domain  of  the  essay 
generator.  Several  types  of  data  are  collected  during  this  session,  on  both 
control  and  experimental  subjects. 

Data  Collection 

The  data  collected  during  the  post- test  were  designed  to  measure  both 
effective  learning  and  self-directed  learning.  Effective  learning  is  defined 
in  terms  of  the  time  required  to  perform  the  task  and  the  number  of  errors 
made  in  performing  it.  For  each  student  data  is  collected  on  the  number  of 
erroneous  attempts  made  to  solve  the  problem  and  the  total  time  taken  to  solve 
the  problem  after  being  presented  with  it.  Self-directed  learning  is  much  more 
difficult  to  measure.  It  was  decided  that  self-directed  learning  is  typified 
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by  two  phenomena:  planning  and  selectivity  in  the  use  of  information  sources. 
The  data  collected  reflect  operational  definitions  of  these  phenomena. 

Planning 

It  is  not  an  easy  matter  to  discover  whether  a student  is  engaged  in 
effective  planning.  One  type  of  data  saved  by  our  PLATO  program  is  the 
sequence  in  which  the  student  accessed  the  information  resources  available 
to  him.  Our  analysis  of  the  troubleshooting  task  presented  to  the  students 
in  the  post-test  session  has  resulted  in  the  formulation  of  a set  of  rules  for 
scoring  deviations  from  the  order  in  which  the  information  sources  should  be 
accessed.  These  rules,  which  we  call  anti -precedence  rules,  take  the  form  of 
prohibitions  of  certain  sequences.  The  extent  to  which  a student  has  departed 
from  sequences  permitted  by  an  ideal  task  analysis  can  be  expressed  in  terms  of 
the  number  of  times  the  student's  study  sequence  violates  the  anti -precedence 
rules. 

Here  is  the  set  of  anti -precedence  rules  based  on  our  analysis  of  the  task 
used  in  the  post-test: 

1.  No  information  source  should  precede  1_ 

2-  i>  2_,  9^  should  not  precede  ^ 

3.  1,  4,  6,  8,  15,  21_  should  not  precede  l^ 

4.  6,  ^ should  not  precede  2 
2»  11  should  not  precede  £ 

2,  11  should  not  precede  8 
2,  11  should  not  precede  12 

(Note:  If  the  student  violates  more  than  one  of  the  rules 
of  #4,  only  one  violation  is  counted.) 

5.  12  should  not  precede  2»  £1 
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Here  is  an  example  of  how  the  scoring  was  done.  Consider  the  following 
hypothetical  sequence  of  accesses  to  information  sources: 

2,  7,  21,  20,  4,  15,  13,  7,  8 

Rule  1 is  violated  once,  because  information  source  Z precedes  7_.  Rule  2 is 
also  violated  once,  because  information  source  2 precedes  source  Rule  3 
is  violated  twice;  information  sources  21^  and  15^  both  precede  13.  Rule  4 is 
violated;  ^ precedes  both  information  sources  15^  and  8.  As  the  note  above 
explains,  this  is  counted  as  only  one  violation.  There  are  therefore  a total 
of  five  violations  of  our  anti -precedence  rules  in  the  example  sequence  shown. 

Selectivity  in  the  use  of  information  resources 

Selectivity  has  to  do  with  the  ratio  of  the  use  of  relevant  information 
sources  to  the  use  of  all  information  sources.  A student  for  whom  this  ratio 
is  high  has  read  primarily  only  relevant  sources.  Three  different  ratios  are 
computed  by  our  program.  The  first  is  the  ratio  of  number  of  relevant  infor- 
mation sources  read  to  total  information  sources  read.  The  second  is  the 
ratio  of  the  number  of  readings  of  relevant  information  sources  to  the  number 
of  readings  of  all  information  sources.  The  third  is  the  ratio  of  time  spent 
reading  relevant  information  sources  to  the  time  spent  reading  all  information 
sources. 


-36- 


Results 


Mean  scores  on  two  measures  for  the  effectiveness  of  the  two  groups 
of  learners  are  presented  in  Table  1.  In  the  final  test  session,  in  which 
students  were  required  to  troubleshoot  a faulty  essay  generator,  those 
students  who  had  not  been  exposed  to  the  training  in  self-directed  learning 
were  slightly  slower  than  those  who  had  received  the  training.  The  experi- 
mental group  subjects,  on  the  average,  solved  the  problem  9 minutes  before 
the  control  subjects.  The  number  of  erroneous  choices  made  by  the  two  groups 
of  subjects  before  identifying  the  appropriate  component  as  defective  was 
about  the  same. 

In  Table  2 the  evidence  concerning  the  selectivity  displayed  by 
students  trained  under  the  two  conditions  is  presented.  The  measures  of 
selectivity  that  are  ratios  of  the  use  of  relevant  information  sources  to 
total  information  sources  show  little  or  no  difference  between  the  two 
groups.  Control  subjects  chose  more  than  twice  as  many  titles  to  read  than 
did  the  experimental  subjects,  suggesting  that  students  in  the  control 
condition  were  not  as  selective;  however,  this  difference  was  not  statisti- 
cally significant. 

Table  3 summarizes  the  measure  used  to  detect  planning.  Planning,  as 
described  above,  is  evidenced  by  few  violations  of  principles  of  efficient 
sequencing  in  reading  the  available  materials.  The  means  suggest  that  the 
experimental  subjects  were  better  planners  than  the  control  subjects,  since 
they  made  only  72%  as  many  planning  violations.  Again,  this  was  not  con- 
firmed statistically. 

A one-way  analysis  of  variance  between  performance  of  the  two  groups 
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Table  1 


Effectiveness  of  Learning  in  Post-test  Session 


Means 

Time  to  complete  Errors 

(minutes)  


Experimental 

65 

2.9 

(n  = 7) 

(31.90) 

(3.13) 

Control 

74 

3.0* 

(n  = 4) 

(25.15) 

(0.00) 

* n = 2 

Standard  deviations  are  in  parentheses. 
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Table  2 


Selectivity  in  Post-test  Session 


Experimental 
(n  = 7) 


Control 
(n  = 4) 


Means 


Titles  P 

Chosen  ^1 


R 


3 


9 

0.73 

0.75 

0.73 

(3.79) 

(0.11) 

(0.09) 

(0.12)  i 

j 

20 

0.83 

0.72 

1 

0.72  1 

(13.89) 

(0.15) 

(0.12) 

(0.08)  ! 

R,  = Ratio  of  number  of  relevant  information  sources  read  to  number  of 
total  information  sources  read 

Rp  = Ratio  of  number  of  readings  of  relevant  information  sources  to 
number  of  readings  of  all  information  sources 

R^  = Ratio  of  time  spent  reading  relevant  information  sources  to  time 
spent  reading  all  information  sources 


Standard  deviations  are  in  parentheses. 
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Table  3 


Experimental 
(n  = 7) 


Control 
(n  = 4) 


Planning  in  Post-test  Session 


Means 

Violations  of  efficient 
sequencing 


Standard  deviations  are  in  parentheses. 
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indicated  that  the  groups  do  not  differ  significantly  on  the  basis  of  time 
to  perform  the  task,  errors  made,  selection  of  relevant  titles,  and  efficient 
sequencing.  The  difference  in  the  number  of  titles  chosen  (Table  2) 
approaches  signigicance,  £ < 0.1. 
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Discussion 


Interpretation  of  these  results  is  problematic.  A cursory  inspection 
of  the  results  leads  one  to  suspect  that  the  special  training  received  by 
the  experimental  group  did  not  have  any  important  effects,  and  is  therefore 
not  a useful  approach  to  take.  Although  the  students  in  the  experimental 
group  seem  to  be  slightly  more  efficient  planners  in  the  post-test  session 
and  slightly  more  selective  readers,  they  don't  seem  to  be  significantly 
more  efficient  learners.  They  made  about  as  many  errors  as  did  the  students 
in  the  control  group  and  they  solved  the  troubleshooting  problem  in  only 
slightly  less  time. 

A closer  examination  of  the  students'  behaviors  in  the  post-test  session, 
however,  reveals  that  the  nominal  experimental  treatment  may  not  have  been 
operational.  The  results  cannot  be  interpreted  as  evidence  that  the  use  of 
the  self-directed  aids  system  is  not  helpful,  because  the  experimental  sub- 
jects were  not  really  using  the  aids  system.  Only  three  of  the  seven  experi- 
mental treatment  students  ever  specified  dependencies  among  information 
sources  that  they  had  chosen  as  relevant.  Only  two  of  them  ever  looked  at 
their  goal  stacks.  None  on  these  students  ever  specified  a dependency  be- 
tween objectives.  A majority  of  these  subjects  (four  of  the  seven)  failed 
to  formulate  more  than  one  objective.  (Those  who  formulated  only  one 
objective  simply  restated  their  task  in  the  form  of  an  objective;  e.g., 
"Identify  the  defective  part  of  the  Essay  Generator.")  Thus,  the  two  groups 
did  not  really  differ  in  functional  treatment. 

In  retrospect,  we  are  impressed  that  the  students  in  the  experimental 
group  were  able  to  do  as  well  as  those  in  the  control  group.  The  control 
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group  students  had  a much  easier  assignment.  They  simply  had  to  make  use 
of  an  automated  table  of  contents  to  read  information  sources  they  thought 
might  help  them  to  solve  the  problem.  What  they  had  to  concentrate  on  was 
learning  new  information  relevant  to  their  task  at  hand,  and  all  their 
intellectual  resources  could  be  devoted  to  this  task.  The  students  in  the 
experimental  group,  on  the  other  hand,  had  a much  more  difficult  assign- 
ment. Not  only  were  they  required  to  solve  the  same  complex  learning 
problems  that  the  control  subjects  had  to  solve,  they  were  also  required  to 
learn  and  use  the  many  details  of  the  very  complex  system  that  was  supposed 
to  aid  them.  Under  these  circumstances,  it  is  surprising  that  the  experi- 
mental subjects  were  able  to  complete  the  task  in  slightly  shorter  times 
than  the  control  subjects,  since  the  experimentals  had  so  much  additional 
apparatus  to  manipulate.  Many  subjects  made  it  clear  to  the  experimenters, 
both  in  verbal  comments  at  the  conclusion  of  the  post-test  and  in  the  written 
critiques  that  were  solicited  from  them  that  they  had  not  fully  internalized 
a set  of  rules  for  the  use  of  the  aids  system  and  that  they  were  very  con- 
fused about  the  functions  of  its  basic  components.  In  fact,  some  students, 
after  muddling  through  a number  of  information  sources,  simply  began  to  make 
wild  guesses  about  which  component  might  be  defective.  In  some  cases, 
students  chose  as  the  defective  component  devices  for  which  they  had  not 
even  read  the  information  sources. 

Not  all  students  who  were  given  the  aids  system  found  it  to  be  useless 
or  even  a handicap,  however.  The  subject  JR,  for  example,  made  very  effective 
use  of  it.  She  showed  good  planning  by  formulating  useful  objectives  and  then 
selecting  information  resources  that  could  help  her  attain  those  objectives. 

By  the  measure  of  planning  discussed  in  the  results,  her  planning  was  perfect; 
she  had  no  violations  of  our  rules  for  efficient  sequencing.  She  was  also  a 
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selective  user  of  information  resources.  She  chose  only  nine  titles  for 
study,  and  her  selectivity  ratios  (explained  in  Table  2)  were  very  high 
(R-j  = 0.89,  R2  = .90,  Rj  = 0.88).  She  was  also  an  efficient  learner.  She 
took  about  an  average  amount  of  time  to  solve  the  problem;  however,  unlike 
many  other  students,  she  made  rm  errors.  She  correctly  identified  the 
defective  component  on  the  first  attempt. 

The  fact  that  this  student  was  better  able  than  others  to  exploit 
the  functions  of  the  automated  aids  system  dramatically  highlights  the 
variation  found  in  student  performance.  An  examination  of  the  standard 
deviations  given  in  Tables  1,  2,  and  3 confirms  this  variation.  The  large 
variation  and  the  small  sample  size  cause  any  differences  between  the  group 
means  to  be  non-significant. 

Perhaps,  then,  the  fault  lies  not  with  the  automated  aids  system  it- 
self, but  rather  with  the  training  program  that  was  designed  to  teach  the 
experimental  subjects  how  to  use  the  aids  system.  A regression  of  scores 
on  the  Nelson-Denny  test  of  reading  ability  on  time  taken  to  complete  the 
task  reveals  an  interesting  difference  between  the  experimental  subjects 
and  the  control  subjects.  This  difference  is  shown  graphically  in  Figure  8. 
Note  that  the  control  subjects  display  the  relationship  that  would  be  expected 
a priori : students  who  score  lower  on  the  reading  test  take  longer  to  com- 
plete the  task.  Experimental  subjects,  on  the  other  hand,  show  considerably 
less  effect  of  reading  ability.  However,  experimental  subjects  scoring  in 
the  low  range  on  the  Nelson-Denny  test  require  much  less  time  to  complete  the 
task  than  control  subjects  scoring  in  this  same  range.  Perhaps  the  automated 
aids  system  benefits  poor  readers  to  a greater  extent  than  it  benefits  good 
readers . 
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VI.  DIRECTIONS  FOR  FUTURE  RESEARCH 


A more  thorough  training  program  is  certainly  called  for.  Students 
in  the  experimental  group  were  exposed  to  a very  large  number  of  aids-system 
functions,  but  were  given  little  opportunity  to  practice  using  most  of  these 
functions.  Each  function  was  demonstrated  once  in  training,  and  the  student 
was  then  required  to  mimic  its  use  once.  For  many  functions,  this  was  the 
only  time  that  the  student  had  to  use  that  function.  In  our  revised  train- 
ing system  for  the  use  of  the  automated  aids  system,  students  will  be  required 
to  practice  with  each  of  the  available  functions  until  the  use  of  each  is 
wel 1 -understood  and  easily  executed  by  the  subject. 

A second  major  problem  in  our  experiment,  in  addition  to  the  lack  of 
adequate  practice  for  those  in  the  experimental  group,  was  the  burden  of 
learning  about  two  complicated  systems--the  automated  aids  system  and  the 
sentence  generator— at  the  same  time.  It  is  very  important  that  subjects 
should  receive  training  drill  on  the  use  of  the  functions  of  the  aids  system 
in  a context  in  which  they  are  not  burdened  with  the  simultaneous  need  to 
puzzle  out  the  workings  of  another  complex  system  at  the  same  time.  To  this 
end,  students  in  future  experiments  will  be  drilled  on  system  functions  in 
the  context  of  simple  learning  problems  first.  Only  after  the  functions 
seem  to  be  well  understood  will  students  be  required  to  use  the  system  to 
solve  the  more  complex  kinds  of  problems  for  which  the  system  was  designed 
to  be  used.  Because  the  system  is  really  designed  to  aid  in  the  solution  of 
complex  problems,  its  use  for  simple  practice  problems  may  seem  superfluous 
to  students.  Our  training  will  therefore  contain  explanations  that  the 
simple  problems  are  used  for  illustration  and  practice.  The  student  will 
be  reminded  that  the  system  is  most  useful  for  the  solution  of  complex 
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problems  in  which  large  amounts  of  information  are  available,  and  that  the 
simple  problems  are  included  only  for  pedagogical  reasons. 

A third  major  problem  in  the  training  our  experimental  subjects  received 
was  that  it  did  not  adequately  motivate  them  to  make  use  of  the  system 
functions.  Three  measures  will  be  taken  to  increase  this  kind  of  motivation 
in  subsequent  experiments.  First,  the  reasons  for  the  inclusion  of  particular 
functions  will  be  explained  more  fully  to  the  subjects.  They  will  be  shown 
how  each  function  can  contribute  to  the  solution  of  a problem  and  under  what 
conditions  the  students  will  have  extra  difficulty  if  he  fails  to  make  use 
of  a function.  Second,  the  students  will  be  induced  to  improve  their  trouble- 
shooting performances.  In  real  world  on-the-job  contexts,  professional 
troubleshooters  understand  that  errors  in  diagnosis  are  expensive.  The 
replacement  of  properly  functioning  components  is  wasteful  of  both  time  and 
materials.  Our  subjects  must  be  induced  not  to  employ  a random  guessing 
strategy  to  identify  the  defective  component  in  a troubleshooting  problem. 

Some  costs  to  the  subjects,  possibly  monetary,  will  be  instituted  in  order 
to  prevent  the  adoption  of  such  a strategy.  On  the  other  hand,  it  is  very 
important  that  subjects  not  be  encouraged  to  be  too  conservative  in  their 
approach.  A troubleshooter's  time  has  value,  and  we  do  not  want  to  drive 
subjects  to  a cautious  study  of  all  the  information  resources  available  to 
them  before  they  make  a judgement.  Indeed,  an  important  part  of  our  con- 
ception of  self-directed  learning  is  that  such  learning  is  selective.  We 
may,  therefore,  find  it  necessary  to  make  use  of  monetary  disincentives  for 
reading  too  many  information  sources.  The  third  measure  we  will  take  to 
increase  the  students'  motivation  will  be  to  institute  some  kind  of  reward 
system  for  the  use  of  certain  functions  of  the  aids  system.  In  particular. 


-47- 


we  would  like  to  reward  the  use  of  those  functions  that  help  the  student  to 
monitor  his  progress  toward  the  accomplishment  of  his  task,  such  as  the 
check-off  and  X-off  functions  of  the  Relevant  Contents  and  Objectives  pages. 
Moreover,  students  would  be  rewarded  for  accessing  the  goal  stack  in  order 
to  plan  a course  of  study.  Ideally,  the  administration  of  rewards  for  the 
use  of  such  functions  should  be  under  the  control  of  the  subject  himself. 

By  following  the  principles  of  behavioral  self-control  set  forth  in  Kanfer 
& Goldstein  (1975),  Mahoney  (1974),  Mahoney  & Thoresen  (1974),  Thoresen  & 
Mahoney  (1974),  and  Watson  & Tharp  (1972),  we  should  be  able  to  help 
students  instill  learning  habits  that  they  can  apply  outside  of  the  experi- 
mental environment  as  well  as  within  it. 
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